EasyExtend.langlets.p4d.langlet.LangletTokenizer

Package EasyExtend :: Package langlets :: Package p4d :: Module langlet :: Class LangletTokenizer |
[hide private]
[frames] | no frames]

# Class LangletTokenizer

               object --+
                        |
         eeoptions.EEShow --+
                            |
    eetokenizer.BaseTokenizer --+
                                |
            eetokenizer.Tokenizer --+
                                    |
                                   LangletTokenizer

Defines langlet specific token stream processor.

Instance Methods | [hide private]
NAME(self, pos, tok, stream) |
Inherited from `eetokenizer.Tokenizer`: `INTRON`, `LEFT`, `RIGHT`, `__init__`, `dedent_to`, `default`, `dot_start`, `multiple_introns`, `post_lex`, `single_intron`, `terminate_stream` Inherited from `eetokenizer.BaseTokenizer`: `add_token`, `set_handler`, `tokenize_file`, `tokenize_fileobj`, `tokenize_string` Inherited from `eeoptions.EEShow`: `maybe_grammar_check`, `maybe_show_cst_after`, `maybe_show_cst_before`, `maybe_show_python`, `maybe_show_scan`, `maybe_show_token` Inherited from `object`: `__delattr__`, `__getattribute__`, `__hash__`, `__new__`, `__reduce__`, `__reduce_ex__`, `__repr__`, `__setattr__`, `__str__`

Class Methods | [hide private]
Inherited from `eeoptions.EEShow`: `show_tokenstream`

Properties | [hide private]
Inherited from `object`: `__class__`

Method Details | [hide private]

### NAME(self, pos, tok, stream) |
Decorators: `@post`